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The phase structure of 2-dimensional topological insulators under a sufficiently strong electron- 
electron interaction is investigated. The effective theory is constructed by extending the idea of 
the Kane-Mele model on the graphenelike honeycomb lattice, in terms of U(l) lattice gauge theory 
(quantum electrodynamics, QED). We analyze the phase structure by the techniques of strong 
coupling expansion of lattice gauge theory. As a result, we find that the topological phase structure 
of the system is modified by the electron-electron interaction. There evolves a new phase with 
the antiferromagnetism not parallel to the direction pointed by the spin-orbit coupling, in between 
the conventional and the topological insulator phases. We also discuss the physical implication 
of the new phase structure found here, in analogy to the parity-broken phase in lattice quantum 
chromodynamics (QCD), known as "Aoki phase". 
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I. INTRODUCTION 

Topological insulators have recently attracting a great 
interest in the field of materials physics 1 -^. They are 
characterized by the gapless modes localized on the sur- 
faces or edges of the system, while the bulk spectrum is 
separated by a finite bandgap. The existence of such gap- 
less modes is ensured by a nontrivial topological invari- 
ant (number) defined by the electron ground state. The 
gapless boundary modes have the properties of massless 
Dirac fermions, which are topologically protected under 
any consistent perturbation or disorder with the symme- 
try. 

The 2-dimensional quantum spin Hall (QSH) insula- 
tor is one of the examples for topological insulators, 
where the shift of the topology is given by the spin-orbit 
coupling acting on the electrons. It was first observed 
in HgTe quantum wells experimentally in 2007 3 , which 
agrees with the theoretical model proposed previously 4 . 
The band structure of QSH insulators is effectively de- 
scribed by the Kane-Mele model, which is based on the 
effective theory of graphene on the honeycomb lattice 5 . 
The system possesses a finite Z2 topological number, 
which gives rise to QSH effect even in the absence of ex- 
ternal magnetic field, related by the so-called Thouless- 
Kohmoto-Nightingale-den Nijs (TKNN) formula 6 -. Its 
topological phase structure is characterized by the com- 
petition between the topological gap from the spin-orbit 
interaction and the non-topological gap from some other 
symmetry breaking effects. Change of the topology oc- 
curs at the phase boundary, where one of the valleys loses 
its bandgap. 

The effect of electron correlation in such an electronic 
system has always been an important problem. Even in 
non-topological Dirac fermion systems, such as graphene, 
it has been proposed that a sufficiently strong electron- 
electron interaction can lead to a spontaneous break- 



ing of some symmetries of the system and a dynamical 
generation of bandgap 7 . In some of the previous stud- 
ies, the idea of quantum electrodynamics (QED), such 
as Schwinger-Dyson equation^, large-TV expansion^iiS, 
exact renormalization group analysi a 11 ' 12 , Monte Carlo 
simulation^ 3 - - — and strong coupling expansio n 16 ' 17 of lat- 
tice gauge theory, has been applied to study the effect of 
electron-electron interaction in graphene (or graphene- 
like) system. It has been predicted that the system can 
show a rich phase structure depending on the pattern of 
symmetry breaking. 

In this paper, we study the effect of a sufficiently 
strong electron-electron interaction on the topological 
phase structure of 2D QSH (topological) insulators. We 
extend the idea of the strong coupling expansion analysis 
on the lattice gauge theory of graphene by adding the ef- 
fect of spin-orbit interaction like the Kane-Mele model. 
In the strong coupling limit of the electron-electron in- 
teraction, there appears an antiferromagnetic (AF) or- 
der spontaneously, and we observe the behavior of the 
order parameter by varying the amplitude of topologi- 
cal and non-topological gaps. As a result, we find that 
the topological phase structure of the system is modified 
from that of the noninteracting system. A new phase, 
which we call here "tilted AF" phase, evolves around 
the phase boundary between the topological and non- 
topological insulator phases, where the direction of the 
antiferromagnetic order is different from that pointed by 
the spin-orbit interaction in the SU(2) spin space. In such 
a phase, we expect that the system can possess a gap- 
less Nambu-Goldstone mode, in contrast to the conven- 
tional topological and non-topological insulator phases. 
We also discuss the analogy between the phase structure 
found here and that of lattice quantum chromodynamics 
(QCD). It is known that lattice QCD with a certain lat- 
tice fermion formalism possesses a parity-broken phase 
similar to the tilted AF phase in the strong coupling re- 
gion, which is called "Aoki phase"—. From the analogy 
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FIG. 1: (a) Configuration of the honeycomb lattice and 
its sublattice structure, (b) Schematic picture of the stag- 
gered magnetic field, (c) Schematic picture of the Kane-Mele 
model. The spin-orbit interaction is introduced in terms of a 
complex hopping between next-to nearest-neighboring sites, 
depending on its direction and spin. 



between these phases, we can give a conjecture on the 
phase structure of topological insulators to some extent, 
from the well-known phase structure of lattice QCD. 

This paper is organized as follows. In Section HH we 
review the band theory and the topological phase struc- 
ture of graphene and topological insulators (Kane-Mele) 
model, in the absence of electron-electron interaction. In 
Section Hill we construct an effective U(l) gauge theory 
on the honeycomb lattice, to incorporate the electron- 
electron interaction in terms of QED. In Section HVl we 
apply the techniques of strong coupling expansion to the 
gauge theory on the honeycomb lattice, and the behav- 
ior of the AF order is investigated in the strong coupling 
limit of the interaction. As a result, we obtain the topo- 
logical phase diagram under the electron-electron inter- 
action, with a new "tilted AF" phase. In Section El 
we discuss the physical properties of the system in the 
tilted AF phase. We also compare this phase structure to 
that of lattice QCD, and summarize the analogy between 
them. Finally, in Section fVTl we conclude our study and 
raise several open questions. 



II. BAND THEORY OF NONINTERACTING 
SYSTEMS 

Before starting the discussion on the electron-electron 
interaction effects, let us briefly review the band theory 
and the topological phase structure of graphenelike sys- 
tems, including topological insulators, without any inter- 
action. The honeycomb lattice consists of two triangular 
sublattices, A and B, both of which are spanned by lat- 
tice vectors Ri = S2 — si and R2 = S3 — Si, with s^i^.3 
the vectors connecting every nearest-neighboring (NN) 
sites (see FigQJa)). The dynamics of electrons on the lat- 
tice is described by the conventional tight-binding Hamil- 
tonian, H T = -t Y,r A , a ,i [ a l{ v A)b a {r A + s t ) + H.c] , 
where a(r^),6(rs) are annihilation operators for A and 
B sites respectively, and the sum by a =t, | assures 
the SU(2) spin symmetry. This Hamiltonian reads 



H T = ~*Eken [6 f (k)$(k)a(k) + H.c] in the momen- 
tum space, so that the eigenvalue is given as E(k) — 
±t|$(k)|, where the momentum kernel $ reads $(k) = 
X)i e * *■ This band structure reveals the well-known 
Dirac cone (valley) structure around two Dirac points 
K± in the Brillouin zone ma. 

One way to open a finite bandgap at the Dirac points 
is an application of a "staggered magnetic field" , Hm = 
m^2 \p?cr z a— b^a z b\, which favors the down spin com- 
ponent at A sites while up at B sites (see Fig[ljb)). This 
term explicitly breaks the SU(2) spin symmetry and the 
sublattice (exchange) symmetry, serving as a mass term 
for the (4-component) Dirac fermion. It opens a bandgap 
\m\ at both Dirac points, keeping the topology of the 
ground state wave function trivial. 

On the other hand, the spin-orbit interaction opens 
a finite gap accompanied with a topologically nontrivial 
ground state. Following Kane and Mele&, it is given in 
terms of a complex hopping term between next-to NN 
(NNN) sites on the honeycomb lattice (see Figfjjc)), 

H SO =t'Y,^^^A)cr z a{v' A )+t , Y,^\v B )a z b^B), 

«rA,r' A )> ((r B ,r' B )) 

(1) 

where the sum is taken over all the pairs of NNN sites 
((rA,r' A )) or ((tb,t' b )). The phase ±i<j> takes the plus 
sign in the direction pointed by the arrows in Figfljc), 
while the minus sign in the opposite direction. If we fix 
the phase <f> = ir/2, the Hamiltonian reads 

H SO = -J2 2i ' Im *2(k) [at(k)a z a(k) - ftt(k) CT ,&(k)] 
ken 

(2) 

in the momentum space, where $2 is defined by ^(k) = 
gik-Rj + e -ikR 2 + e tk-(R 2 -Ri)_ Similar to the staggered 

magnetic field, it breaks the sublattice and spin symme- 
try, while it also breaks the exchange symmetry of two 
valleys in addition. Thus, it opens a bandgap with an am- 
plitude |3v3Y| for each valley as an "effective mass" for 
Dirac fermion, with its sign depending on the valley /spin 
indices. As a result, the ground state wave function ac- 
quires a nontrivial topology with a non-zero Z2 topolog- 
ical number given in the momentum space, leading to a 
quantized spin Hall conductivity. 

The topological phase structure of the system is char- 
acterized by the competition between the conventional 
gap and the topological gap. Applying both gap opening 
effects given above, one valley obtains a bandgap with 
amplitude m + , while the other m — 3V3t' . When 
the conventional gap is dominant over the topological one 
(\m\ > |3-\/3t'|), both valleys obtain a bandgap with the 
same sign, leaving the ground state trivial. When the 
spin-orbit interaction is dominant (|m| < |3-\/3V|), the 
system behaves as a topological insulator. Two phases 
are separated by a line m = ±3-\/3V , on which one of the 
valleys loses its gap while the other remains gapped. In 
this paper, we focus on how the electron-electron inter- 
action alters such a topological phase structure. 
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III. LATTICE GAUGE THEORY DESCRIPTION 



formalism as 



Here we assume that the electron-electron interaction 
is mediated by the electromagnetic field, namely U(l) 
gauge field ^1=0,1.2,3- The dynamics of fermions can be 
reconstructed in terms of an imaginary time action on 
the honeycomb lattice (see Appendix lAl for details), 



S F =\ E [aHr A ,T)U (r A ,T)a{r A ,T + At) - H.c] 

+ \ E [b\v B ,r)U {t B ,T)b{v B ,T + At) - H.c] 
+ E [a t (rA,r)b(rA + s i ,r)+H.c.] , (3) 



with 
exp 



the U(l) link 

hA 



: /™dT'A (r,T') 



variables Uq(t, t) = 
Dynamics of the spa- 
tial components of electromagnetic field, Ui (or Aj), 
is neglected, which is referred to as "instantaneous 
approximation"—, since the gauge field propagates 
much faster than the fermions, so that the retardation 
effect becomes considerably small. Here the imagi- 
nary time direction is rescaled by the Fermi velocity 
v F to avoid the space-time anisotropy in the Dirac 
operator, and is discretized by the lattice spacing At 
comparable to the spatial lattice constant a = \si\. 
Since this discretization leads to a pair of fermion 
doublers in the temporal direction^, here we suppress 
the spin index a to attribute the doublers to the 
realistic spin degrees of freedom, like in the staggered 
fermion formalism- 1 !--. The lattice action Eq.Q is 
invariant under the global U(l) charge transformation 
a^e l9 a, a^a^-* 9 , b ->■ e i9 b, fet &t e -«, and 
the U(l) spin transformation 



a c -> e i9 a e , -> aje l9 ~, b c -> e~ l9 b Cl b\ -> &t e -rf 
a -> e- i§ a , a t _> at e -* b -> e i§ b , tf Q -+ &t e * (4) 



where the label e/o represents whether the discretized 
imaginary time t/At is even or odd. It should be noted 
that spin SU(2) symmetry is broken down to U(l) sub- 
space due to the temporal discretization, which is anal- 
ogous to the intrinsic flavor (taste) symmetry breaking 
in the staggered fermion formulation. Similarly to the 
staggered fermion, the full spin symmetry is restored in 
the continuum limit (t/At — > 0). Since we can choose 
the spin direction arbitrarily, here we define the remnant 
U(l) spin symmetry in the (2, z)-plane, generated by the 
spin operator a y . 

The spin-orbit interaction term Hso and the staggered 
magnetic field Hm are reconstructed in the path integral 



&so =t — 



E 



±ia^(r A ,T)a(r' A ,T) 



At 
&m =m — 



^2 ±lb\v B ,T)b{v' B ,T) 

(('fl/ B )),T 



(5) 
(6) 



where both terms break the sublattice symmetry and the 
remnant U(l) spin symmetry explicitly. Hereafter we 
suppress the rescaling factor At/v f for simplicity, so that 
we regard the parameters t, t' and m dimensionless. 

Dynamics of the gauge field can also be defined on 
the honeycomb lattice, in terms of the polynomial of 
link variables Uq. The gauge kinetic term Sq is pro- 
portional to the parameter j3 — eov F /e 2 , which corre- 
sponds to the inverse of the (effective) Coulomb coupling 
strength. When the Fermi velocity of the electron v F is 
sufficiently small compared to the speed of light, the ef- 
fective coupling a = e 2 /47reo« F becomes larger than the 
usual q;qed = e 2 /47reoc ~ 1/137, since a slower electron 
feels the effect of the electromagnetic field more strongly. 
The parameter ft becomes quite small in such a system. 
For instance, ft ~ 0.04 in vacuum-suspended graphene, 
where v F is about 300 times smaller than the speed of 
light. Sg vanishes in the strong coupling limit ft = 0, 
i.e. the spatial propagation of the electromagnetic field 
is completely suppressed. 



IV. STRONG COUPLING ANALYSIS 

A. Antiferromagnetism in the non-topological 

system 

Let us first review the behavior of the non-topological 
system, in the absence of the spin-orbit interaction and 
the staggered magnetic field, in the strong coupling limit 
ft = 0. In this limit, we can rewrite this effective action 
only in terms of fermionic field variables by integrating 
out the gauge degrees of freedom, 

SP=tJ2 [aHr A ,T)b(r A +s i)T ) + -tt.c.] (7) 

rA-A,T 

E n ™ ( T ^A ( T + At ) + E Tlr « ( T ) n rB + Ar ) 



LrA,r 



where 



At) 



= nt 



ai(r A ,T)a(r A ,T) and 



At) 



bA?B,T)b(rB,T) denote the local charge density at time 
t. In the leading order of the strong coupling expan- 
sion, an on-site interaction with a temporal lattice spac- 
ing is extracted from the long-range Coulomb interaction, 
which is similar to the on-site repulsion term in the phe- 
nomenological Hubbard model. 
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Respecting the sublattice symmetry and the remnant 
U(l) spin symmetry, here we take the mean-field ansatz 



(a t (r j4 ,r)a(r j4 ,T)) = \ <j x - o 2 



(&t(r fl ,r)6(rB,r)) 



(8) 
(9) 



where 01.2 are real values. (One should not confuse the 
mean fields <ji,2 with the Pauli matrices &x,y,z- ) Thus, 
by integrating out the fermionic field variables, we obtain 
the thermodynamic potential (free energy) of the system 
per a pair of A and B sites, 



F eS (a) 



1 



d 2 kln 



4 



|f$(k) 



(10) 



where the momentum integration within the Brillouin 
zone Vi is normalized as J Q d 2 k = 1. Here the order 
parameter a = g\ + ia 2 appears only in the form of 
| (7 1 2 = af + a 2 , which reflects the remnant U(l) spin sym- 
metry, d — > <re 2ie . This symmetry gets broken sponta- 
neously when \a\ takes a finite expectation value, but the 
phase of a can be chosen arbitrarily unless the symmetry- 
breaking source Sso or Sm is applied. The arbitrariness 
results in the emergence of gapless Nambu-Goldstonc 
boson when the U(l)v symmetry is spontaneously bro- 
ken, a serves as the order parameter for the spontaneous 
breaking of the sublattice symmetry and the remnant 
U(l) spin symmetry, which corresponds to the spin den- 
sity wave (SDW) order, or the antiferromagnetism, on 
the honeycomb lattice. 

The first term in Eq. ([T0l) . which comes from the tree 
level of the bosonic auxiliary field a, becomes dominant 
for | a | — > oo, while the second term, which stems from the 
fermion one-loop effect, yields a logarithmic singularity 
around a = 0. Therefore, F c g(a) possesses a minimum 
at finite \a\, so that there appears a spontaneous an- 
tifcrromagnetic order with an arbitrary direction in the 
remnant U(l) spin space. This antiferromagnetism opens 
a finite bandgap in terms of a dynamical "mass term" , 
while the Z 2 topology of the system remains unchanged. 



B. Effect of interaction in the topological system 

Let us now investigate how the spontaneous antiferro- 
magnetism obtained above behaves in the presence of the 
(Kane-Mele-type) spin-orbit interaction Sso- We also 
introduce the uniform staggered magnetic field Sm for 
convenience of later analysis. Since the spin-orbit inter- 
action and the staggered magnetic field explicitly break 
the remnant U(l) spin symmetry in the <j\ -direction, the 
order parameter components <j\ and a 2 should be distin- 
guished here. Here the effective potential in the strong 
coupling limit reads 



^eff(<T) 



(cti - 2m) 2 



4 



[ d 2 k\n [£(ai,a 2 ,t';k)} 
Jn 



(11) 



°2 




0.00 0.01 0.02 0.03 0.04 
t'/t 



0.05 0.06 



FIG. 2: The behavior of the order parameter o-i as a function 
of the amplitude t' of spin-orbit interaction, in the absence of 
explicit mass m. 02 vanishes at the critical value t' c = 0.0538t. 



where we have shifted the order parameter <7i/2 + m — > 
(Ti/2, which serves as the "modified" source m due 
to the electron-electron interaction. £ (a±, 02, t'; k) = 

yVi/ 2 " 2i'Im$ 2 (k)] 2 + (cr 2 /2) 2 + |i$(k)| 2 denotes the 
energy of an electron in the conduction band. The min- 
imum of this effective potential (<7i, 02) satisfies the gap 
equations 



dF, 



cir 



dF eB 



= <7i — 2m — / cPk 

(£1,0-2) J n 



(5-l,cr 2 ) 



02 



d 2 k- 



CTi/2-2i'Im$ 2 (k) 
<7 2) i';k)] 2 

2 /2 



= 



[£(ai,a 2 ,t';k)] 



(12) 
0. (13) 



Since the potential is even in cr 2 , the solution 
should satisfy either <7 2 = 0, where the antifer- 
romagnetic order is aligned in the <7i-direction. or 
(5-1, cr 2 ) — 0, where it is tilted toward 

the CT 2 -direction. 

First, we consider the case in the absence of the stag- 
gered potential m, where the effective potential becomes 
even both in <j\ and cr 2 . In this case the potential mini- 
mum satisfies <j\ = 0, so that the antiferromagnetic order 
points the <7 2 -direction, i.e. it is confined in the xy-plane 
(see Appendix [Bl for detail). Here it should be noted that, 
even though the spin-orbit interaction explicitly breaks 
the remnant U(l) spin symmetry in the u\ -direction, the 
antiferromagnetic order is aligned in the a 2 -direction, or- 
thogonally to G\ . 

The quantitative behavior of a 2 is obtained by mini- 
mizing the effective potential 



F ef{ (a 2 ) = ^- [ d 2 kln[£(0,a 2 ,;';k)f 
1 Jn 



(14) 



by cr 2 . Since the finite bandgap 3^/3t' from the spin-orbit 
interaction at each Dirac point moderates the logarithmic 
singularity in the loop integral, it suppresses the expec- 
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FIG. 3: Behavior of the order parameters 0x2 as a function 
of m, with the amplitude of spin-orbit interaction fixed at 
t' = 0.5t' c . The staggered magnetic field m monotonically 
enhances <ti as its external source, while it suppresses the 
orthogonal order parameter 02- 




FIG. 4: The path of order parameter (0-1,02), with the spin- 
orbit interaction t' fixed and the staggered magnetic field m 
varied. The path starts from 01 = at m = 0, and evolves 
to <7i — > 00 monotonically as m — > 00, since the staggered 
magnetic field m serves as the external source for the antifer- 
romagnetic order in the 01-direction. 



tation value of 01 . Second order phase transition occurs 
at the critical value t' c — 0.0538i, as shown in FigJ5J 

Next, we fix the spin-orbit interaction t' and introduce 
the uniform staggered magnetic field Sm- For instance, 
by fixing £' = 0.5^, we can observe the behavior of the 
order parameters a± t 2 as a function of m, as shown in 
FigEl Since Sm serves as a source term for the AF order 
in the cri-direction, it eventually tilts the direction of a 
from the 02-axis toward the cri-axis. Path of the solution 
(f 1, 02), with t' fixed and to varied, is displayed in Fig|4] 
(see Appendix [Cl for details). Starting from ai = at 
m = 0, 01 monotonically increases as a function of m. 



FIG. 5: The phase diagram of this system in the (t' , <ri)-space. 
Here we use the "modified" mass ai instead of the bare mass 
(staggered magnetic field) m, since there is a unique one-to- 
one correspondence between m and 01 , depending on t' . As a 
consequence of the interplay between the electron-electron in- 
teraction and the spin-orbit interaction, there appears a new 
"tilted antiferromagnetic (AF)" phase (02 7^ 0), between the 
normal AF phase (02 = and <7i/2 > 3v3t') and the topo- 
logical phase (<T2 = and <j\/2 > 3\/3Y). 




FIG. 6: The phase diagram in Fig[5] mapped in the (t',m)- 
plane. The dashed line corresponds to the topological phase 
boundary 5i/2 = 3\/3t' • The tilted AF phase converges to a 
point at (t ,m) — (0,0). 



For t' < t' c , the AF order gets tilted from <72-axis to 
the ui-axis, and finally 01 vanishes at some critical value 
of m depending on t' . On the other hand, if t' > t' c , 
the path starts from o\ = 02 — 0, and <j\ evolves with 
the staggered magnetic field m. In both cases, when 
m reaches a sufficiently large value, the electron-electron 
interaction can be neglected compared to the explicit gap 
to, so that the AF order a is aligned in the tri-direction, 
parallel to the spin-orbit interaction and the staggered 
magnetic field. 



Thus we can map the phase diagram of this system 
in the parameter space (t',m). Since o\ monotonically 
increases as a function of m (depending on i'), here we 
take a i as a control parameter instead of m. There are 
two branches of phase boundary between the phase with 
finite 02 and that with 02 — 0: one is present for the 
whole range of t', while the other is restricted in the re- 
gion t' > t' c . When the explicit bandgap from t' and m 
is extremely large compared to the scale of the electron- 
electron interaction, the logarithmic singularity in the 
fermion loop integral becomes dominant only around the 
the topological phase boundary a\/2 = 3y/3t', where 
one of the Dirac cones loses its bandgap. Therefore, the 
(T2 7^ phase shrinks to the topological phase boundary 
in this limit, so that the phase boundaries characterized 
by o"2 discussed above approach asymptotically along the 
topological phase boundary. As a consequence, the phase 
structure of this system is classified into three phases, as 
shown in Fig(5j 

• "Topological" phase (01/2 < 3\^3t' and 02 = 0): 
The AF order, aligned in the a% -direction, is rather 
small compared to the explicit gap given by the 
spin-orbit interaction. Thus, the system becomes a 
Zi topological insulator, yielding the quantum spin 
Hall effect even under the electron-electron interac- 
tion. 



• "Normal AF" phase (01/2 > 3^3*' and 02 = 0): 
The commensurate AF order <j\ exceeds the ex- 
plicit gap given by the spin-orbit interaction t', 
so that the system becomes a conventional (non- 
topological) insulator. 



• "Tilted AF" phase (a 2 ^ 0): The AF order is tilted 
from di-axis toward the 02-axis. 

One should be careful of the tilted AF region at if — 0. 
This region can be reached at if = m = 0, where the 
remnant U(l) spin symmetry is not explicitly broken. 
Since (01,02) can be chosen arbitrarily with keeping 
|cr| 2 = crj + erg constant, here the ground state can take 
any point within this region. If we map it by the original 
parameter set (t',m), as shown in FigjSl such a region 
corresponds to the original point of the phase diagram. 

As a result, in the presence of the electron-electron in- 
teraction, the phase structure of the system is altered 
from the noninteracting system, with the emergence of 
"tilted AF" phase between the topological phase and the 
conventional insulator phase. In other words, the topo- 
logical phase "boundary" in the noninteracting system 
evolves into the tilted AF "region" by the effect of the 
electron-electron interaction. We shall discuss the phys- 
ical properties of this phase in the next section. 
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FIG. 7: Schematic picture of the order parameters derived in 
this study, ai and 02 are antiferromagnetic (AF) orders cor- 
responding to two directions in the remnant U(l) spin space, 
which we denote M z and M x here. The staggered magnetic 
field m explicitly breaks this symmetry to the cri-direction. 
When (a 2) 7^ 0, the AF order is tilted to the 02-direction 
to some extent. If we extend this argument to the full SU(2) 
spin space, another direction M y is restored, so that the tilted 
AF acquires U(l) degree of freedom in choosing its direction, 
which may result in a massless Nambu-Goldstone mode. 

V. DISCUSSION 
A. Physical properties of the tilted AF phase 

What does the emergence of the "tilted AF" phase 
physically imply? We should recall that the order param- 
eters <j\ and <72 correspond to the AF order in the z- and 
x- (or j/-)directions respectively, breaking the remnant 
U(l) spin symmetry. In the tilted AF phase, the direc- 
tion of the AF order is tilted from z-axis, pointed by the 
spin-orbit interaction and the staggered magnetic field, 
to the xy-plane, by the interplay between the electron- 
electron interaction and the spin-orbit interaction, as 
sketched in FigfTl In the absence of the staggered mag- 
netic field m, is completely tilted to the 02 -direction, 
which is consistent with the "XY-antiferromagnetic in- 
sulator" phase found in the analysis of the Kane-Mele- 
Hubbard model22~— . 

If we restore the spin space from remnant U(l) to full 
SU(2), there appears a U(l) degree of freedom in choos- 
ing the direction of the AF order within the rry-plane, 
which results in the appearance of a gapless Nambu- 
Goldstone mode, while the fluctuation in the z-direction 
becomes massive. Thus, although the fermion spectrum 
is gapped, it is possible that such a gapless boson may 
carry an electric current, turning the system back from 
insulator into a (semi-)metal. On the other hand, when 
the antiferromagnetic order is aligned in the z-direction 
("normal AF" phase), the phase fluctuation of the order 
parameter will result in two massive modes, so that the 
system remains an insulator. Phase transition between 
the topological insulator (QSH phase) and the metallic 
phase has also been suggested in quantum Hall systems, 
driven by a disorder—. 

In the phase diagram obtained here, the phase transi- 
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tion between the topological phase and the conventional 
insulating phase occurs without closing the bandgap of 
fermions. This behavior appears to contradict the previ- 
ous studies in the noninteracting Dirac fermion system, 
where the gap closing is essential for the topological phase 
transitio n 27 ! 28 . In this study, however, one should note 
the appearance of a gapless NG mode in the tilted AF 
phase, which is not taken into account in the noninter- 
acting system. The qualitative properties and physical 
effects of this NG mode, which cannot be investigated 
within our analysis due to the restriction of the spin space 
as a lattice artifact, remains a future problem. 



B. Analogy with the phase structure of lattice 
QCD 

The phase structure of 2-dimensional topological insu- 
lators observed here can be understood in analogy with 
that of lattice QCD, which has been thoroughly studied 
by strong coupling expansion, Monte Carlo simulations, 
etc. In order to avoid the doubling of quarks arising 
from the lattice discretization, one can take the Wilson 
fermion formalism, where a momentum-dependent mass 
term (Wilson term) is employed in addition to the uni- 
form mass term, to shift the degeneracy of doublers^. 
One can extract a single species of fermion with the 
lowest effective mass out of the doublers, by taking the 
Wilson parameter sufficiently large. This mechanism is 
analogous to the spin-orbit interaction on the honeycomb 
lattice, which shifts the degeneracy of two valleys in the 
Brillouin zone fi. Of course, the continuous chiral sym- 
metry of the quarks, which corresponds to the remnant 
U(l) spin symmetry in our graphene model, is explicitly 
broken by this effective mass term. 

It is known that lattice QCD with the Wilson fermion 
formalism has a characteristic phase structure. In the 
strongly coupled QCD, the chiral symmetry is sponta- 
neously broken with a finite chiral condensate (ipip), dy- 
namically generating a mass of quarks. The mass term 
rmpip serves as a source term for the chiral condensate. 
In lattice QCD with a single-flavor Wilson fermion, on 
the other hand, there appears a finite pion condensate 
(^75^), which is orthogonal to the chiral condensate 
(tpip) in the chiral symmetry spaced. This phase is 
called "Aoki phase" , where the parity symmetry is spon- 
taneously broken by the pion condensation. In the two- 
flavor theory, the Aoki phase is characterized by a neutral 
pion condensate (^(75 ® T z )tp), where the parity-flavor 
symmetry is broken instead of the parity symmetry it- 
self (r z is a Pauli matrix with respect to the "isospin" , 
corresponding to the flavor of quarks). If we fix the Wil- 
son parameter and give a sufficiently large mass term 
uniformly to all the doublers, the pion condensate dis- 
appears, and only the chiral condensate acquires a finite 
expectation value. It is known that the transition be- 
tween the Aoki phase and the normal phase is a second 
order phase transition around the strong coupling limit. 



Normal AF 



Normal AF 




(ii) t' > t' c 



FIG. 8: The schematic phase structure of graphenelike system 
with the spin-orbit interaction, conjectured in analogy to that 
of lattice QCD with the Wilson fermion formalism. Here we 
fix the spin-orbit interaction t' , and vary the electron-electron 
interaction strength g 2 and the uniform mass (staggered mag- 
netic field) term m. From the strong coupling limit (g 2 = 00) 
investigated in our study, the phase structure evolves to the 
weak coupling region differently, depending on whether t is 
below or above the critical value t' c — 0.0538t. 



On the other hand, in the weak coupling regime, the Aoki 
phase is split into several cusps and shrinks to the poles 
corresponding to the doublers; between these poles, the 
topology of the system remains non-trivial as in the free 
Hamiltonian. 

Since the spin-orbit interaction on the honeycomb lat- 
tice and the Wilson term of lattice fermions have the 
similar topological features as we have seen, we can re- 
late the "tilted antiferromagnetic (AF)" phase observed 
in 2-dimensional topological insulators here to the Aoki 
phase in the lattice QCD with the Wilson fermion with 
the following analogy (see Table [I]). The tilted AF phase 
is characterized by the nematic AF order (0-2) ^ in the 
U(l) remnant spin symmetry space, while the Aoki phase 
is characterized by the pion condensation {ipi^^ip) 7^ in 
the continuous chiral symmetry space. In both cases, 
the emergent order parameter is orthogonal to the di- 
rection pointed by the external source, namely the stag- 
gered magnetic field Sm and the mass term rmpip re- 
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Kane— Mele model 


Lattice QCD 


Non-topological source 


staggered magnetic field (m) 


mass term 


Topological source 


spin-orbit interaction (£') 


Wilson term 


Splits the degeneracy of 


valleys 


doublers 


Explicitly breaks 


remnant U(l) spin symmetry 


continuous chiral symmetry 


Interaction is mediated by 


photons (electromagnetic field) 


gluons 


Induced phase 


tilted AF phase 


Aoki phase 


Order parameter 


(o"2)(~ (a'axd — b' <Jxb)) 


{^pi^/5ip) (pion condensation) 



TABLE I: Analogy between the Kane-Mele model (2-dimensional topological insulators) and lattice QCD with Wilson fermion. 



spectively. Relying on this analogy, we can make some 
conjecture about the phase structure of the graphenelike 
system from strong coupling to weak coupling, as shown 
schematically in Fig[8] 

• (i) When t' < t' c , the system reveals the tilted AF 
(Aoki) phase around m = in the strong coupling 
limit. Going down to the weak coupling region, 
this phase splits into two branches, and shrinks to 
two points corresponding to the topological phase 
boundary in the free limit. The topological (QSH) 
phase with a finite spin Hall conductivity appears 
when the coupling strength reaches a sufficiently 
small value. 

• (ii) When t' > t' c , the system reveals the topologi- 
cal phase around to = even in the strong coupling 
limit, while the tilted AF phase appears at inter- 
mediate to. Here the tilted AF phase is already 
split into two branches, corresponding to the sign 
of mass. Going down to the weak coupling region, 
the tilted AF phase shrinks to two points corre- 
sponding to the topological phase boundary. 

Here we have assumed that the phase boundary is con- 
tinuous from the strong coupling limit to the free regime. 
The phase transition around the strong coupling limit is 
second-order as shown in our study, while it remains un- 
clear in the weak-coupling region whether it is first-order 
or second-order, as it has been argued in lattice QCD 30 . 

VI. CONCLUSION 

In this paper, we have observed the effect of the 
electron-electron interaction on the phase structure of 
2-dimensional topological insulators, extending the idea 
of previous works with respect to the spontaneous sym- 
metry breaking in graphene. Kane-Mele model on the 
honeycomb lattice is extended by introducing the effect 
of electron-electron interaction mediated by electromag- 
netic field (U(l) gauge field). By the techniques of strong 
coupling expansion of lattice gauge theory, we have ob- 
served the behavior of the spontaneous antiferromagnetic 
(AF) order in the strong coupling limit of the interac- 
tion. As a result, we have found that the topological 



phase structure is modified from that of the nonintcr- 
acting system, by the emergence of a new "tilted AF" 
phase in between the normal insulator and the topologi- 
cal insulator phases. The AF order is not parallel to the 
direction pointed by the spin-orbit interaction and the 
staggered magnetic field in the spin SU(2) space in this 
phase, which will result in the emergence of a gapless 
Nambu-Goldstone mode corresponding to the in-plane 
rotation. We have also shown the analogy between the 
phase structure of topological insulators shown here and 
that of the strongly coupled lattice QCD with the Wil- 
son fermion formalism. The tilted AF phase is similar to 
the so-called "Aoki phase" in lattice QCD in that both of 
them are characterized by an order parameter orthogonal 
to the external source term in the continuous symmetry 
space. Such an analogy may help us understand the be- 
havior of topological insulators with an electron-electron 
interaction from strong coupling to weak coupling regime. 



There remain several open questions to be solved. One 
is the restriction of the SU(2) spin symmetry space down 
to U(I), due to the lattice discretization in the imaginary 
time direction. Such a lattice artifact obscures the details 
of the NG mode appearing in the tilted AF phase, so that 
its quantitative effect on the charge/spin transport prop- 
erties is left to be calculated. It also excludes the spin 
singlet orders, such as a charge density wave and Hal- 
dane flux state, which induces quantum anomalous Hall 
effeci£^-~— . Such effects should be taken into account by 
the models with an exact symmetry structure, such as 
the extrapolation to a (hypothetical) multiflavor theory, 
or by other techniques that do not require any lattice 
regularization process. The relation to the physics in the 
realistic topological insulators would be another problem. 
Comparison of our findings to the similar strong coupling 
analysis in the effective model of 3-dimensional topolog- 
ical insulators 3 -!, such as Bi2Se3 or Bi2Te3, would give 
us some clues. The interaction effect on the quantum 
Hall states and the gapless surface (edge) states would 
be experimentally of a great importance. 
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Appendix A: Construction of lattice action 

Here we provide a rigorous description about the 
derivation of path integral formalism on the honeycomb 
lattice, and give a physical interpretation to the doubling 
problem arising from the lattice discretization. In the 
Hamiltonian formalism, the dynamics of electrons and 
the electromagnetic field is given by the Hamiltonian 



^ = t EE E s[at(r A )Ui(r A ,0)bt(r A + Si )+-H.c. 

(Al) 



t a s=±i=l,2,3 



^E^E^(^)] 2 + ! E [*.(*>*)] s 



2 ^ | ^ L-.v-~> -/j ■ 4 

zGaZ I TA,i rEAuB 



(A2) 



where the gauge field operator A and its conjugate mo- 
mentum (electric field) E are defined on the lattice links 
as 



Ai(r A ,z) = - dr ■ A(r, z), 

J r A 
^ /TA+S, 

Ei(r A ,z) = - dr ■ E(r, z) 

a 



A z {r,z) 



I 1M 

^ r-z+a 

a Jz 

z+a 



dz'A(r, z'), 



E z (r,z)=- f dz'E(r,z'), 
a Jz 



(A3) 
(A4) 
(A5) 
(A6) 



and P is a plaquette operator constructed of link vari- 
ables Ui = exp[ieaAi] and U z = exp[ieaA z ]. The sum- 
mation J2 P is taken over all the plaquettes (both honey- 
comb and square) on the lattice. It should be noted that 
the gauge field operators are defined in the 3-dimensional 
space (r, z). As for the fermions, we have performed a 
Bogoliubov transformation defined by 



0+ — > a+, a_i — > a_ 



h^bl, 6|~>fe-, (A7) 



where we should note that the labels f and J. represent the 
eigenvalue of an arbitrarily chosen spin operator. Here 
we take it as the Pauli matrix a y for later convenience. 



By splitting the inverse temperature f3 by an infinites- 
imal timeslice At as 



Z = Tie- " = Tr [< 



-AtH -AtHI 



(A8) 



and inserting complete sets of states between every 
e~ , we obtain the Euclidean action on the honey- 
comb lattice, 



S F =Ai 



^ ald T a s + 2J b\d T b s 



Ar^i/ F (r), 

T 

(A9) 



Sa = 



Via 3 At 



2 

At 



J2 {drAtf + jY^idrAzf 



(A10) 



where the derivative d T is defined as d T f{r) = [/(r + 
At) — /(r)]/Ar. If we go back to the original spin rep- 
resentation, Eq. (|A9[) reads 



S F =Ar ]T [al&raa + (AT)(d T a\)(d T a±) 

TA,T 

+ At 



(All) 



Y, [btd T b a + (Ar^drbl^d^)] +Ar^ff f (r 



Therefore, the lattice action does not preserve the global 
spin SU(2) symmetry unless the continuum limit At — > 
is taken, but it is still invariant under the remnant U(l) 
rotation generated by a y . 

Here we fix the lattice anisotropy a / At = v F , to re- 
produce the ratio between the intrinsic cutoffs of energy 
and momentum given by the Dirac cone structure. If 
we take the physical value in monolayer graphene, the 
ratio reads At /a — v^ 1 3> 1, so that we can apply sad- 
dle point approximation to the second line of Eq. (jAlOl) , 
yielding 



P = l i.e. VxA = 0. 



(A12) 



Thus we can take the scalar potential (f>(r, z,t), which 
satisfies the relations 

Ui(r A ,Z,T) — e ieaA i( r A,Z,T) _ e i[4>(T A +Si,Z,T)-4>{r A ,Z,T)\^ 

(A 13) 



U z {t,z,t) =e 



ieaA s (r : z,r) 



i[(f>(r,z+a,r) — 4>(r,z,r)] 



(A14) 



This approximation drops off the retardation of the elec- 
tromagnetic field, which is referred to as "instantaneous 
approximation" . Such an approximation enables us to re- 
construct the gauge action only in terms of its temporal 
component. By the local gauge transformation 

a s (r,T) -► e-^°^a s (r : T), b s (r, T ) ->■ e ts ^°^b s (r,T) 

(A15) 
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and taking a new link variable 

U (r,Z,T) EE e -i[^(w+AT)-«w)] = e -te(r,«,T) ) 

(A16) 

we can set the spatial link variables to unity. Due to this 
definition, the Polyakov loop obeys the constraint 

1[U (t,z,t) = 1, i.e. ]T0(r,z,r) =0. (A17) 



As a result, the lattice action reads 

Sf =At £ a t(r A , r) )-Mr,,r) 

+ Ar^if f (r) (A18) 



5 G = 



a 



2e 2 At 



5^ [0(r A ,z,T)-e{r A + Sl ,z,T)Y 



3V3 a 

^2"A7 



^ [0(r,z + a,r)-0(r,z,r)] ; 



r£AuB,z,T 



(A19) 



The kinetic term of the gauge field in Eq. (|A19j) is given in 
the non-compact form. In the continuum limit (At — > 0), 
it can be regularized by the compact form 

S G = -V3f3 J2 Re[U (r A + Si ,z,T)UZ(r A ,z,T)] 
-3V3/3 Re[t/ (r,z + a,T)t/ *(r,2,T)]. 

r£AUS,2,T 

(A20) 



Let us prove that the fermionic action in Eq. (|A18l) is 
equivalent to the "staggered fermion" formalism, which 
has been naively given in Eq.Q. Since a s and b s are not 
operators but just Grassmann variables, we can propose 
the following change of integration variables in the path 
integral: 

<2+(t) — > ol{t), a +( T ) <5(t + At'), 
o_(t) — >• a(r), o,-(t) — >• a(r + At') 
6+(t)^/3(t), ;(t)^/3(t + At'), 
b-(r)->P(T), & + _(t) j8(r + At'), (A21) 

with a finer time-slice At' ee At/2. The spin degrees of 
freedom are absorbed in the temporal lattice mesh. The 
anti-periodicity in the temporal direction 

a(r' + p) = -a(r'), p(r' + p) = -p(r') (A22) 



also holds for the new fermionic fields. By this transfor- 
mation, the fermionic action reads 

S F = J2 [aVod - a'Vja] + £ [pV p' - P'V^P 
- It At' ^ [&P + j3a] , (A23) 

rA,i,~r' 

where \' = \{ T ' + At') for \ = a, /?, a, p. The new link 
variable Vq is defined by 

„, fl (T'/AT' = even) 

V ^ T) = {u (r,r'-Ar') \r>/Ar> = odd) ' ^ 

This fermionic action is invariant under the global rota- 
tion corresponding to the remnant U(l) spin symmetry 
defined in Eq.(|4]). It agrees with the staggered honey- 
comb lattice action in Eq.@, if we renormalize a and 
P by the factor l/v2 and interpolate Vb. c by a dynami- 
cal U(l) link variable. Therefore, the change of variables 
in Eq. (|A21[ ) gives the correspondence between the true 
spin degrees of freedom and the staggered fermions on 
the honeycomb lattice. 



Appendix B: Absence of order parameter o\ at m = 

In this appendix, we show how the order parameter a\ 
gets suppressed in the absence of the staggered magnetic 
field to, by solving the gap equations (fT2j) and (fT3|) . 

Let us assume the solution b\ ^ 0. Since the potential 
F e g is even both in o\ and cr 2 at m = 0, we can set 
a \ > without losing generality. The solution of Eq. (fT3|) 
is twofold: 

(i) If 02 = 0, the convexity around (01,02) m < J 2- 
direction reads 



d 2 F cS 



da 2 



= 1 



d 2 k- 



1/2 



1 

01 



o-i 



ai/2 



(Bl) 



d z k- 



1 



= __ d z k 



a i Jken 



2t'Im$ 2 (k) 
"[£(*!, 0,t';k)] 2 ' 



where we have used Eq. (fT2")l . Since $(— k) = $*(k) 
and $ 2 (— k) = $*(k), we can separate the Brillouin 
zone fl into two regions A± corresponding to the sign 
of 2i'Im<j> 2 (k), which yields 



d 2 F cff 



da 2 



-Y. 



o-i 



d 2 k- 



A± 



2t'Im$ 2 (k) 
[£(a u 0,t';k)] 



(B2) 



— f d 2 k> 2t ' Im$2 ( k ) , -2t'Im$ 2 (k) 



CT i Ja + 



[£(oi,0,t';k)] 2 [f(cri,0,-t';k)] 2 



Since 5\ > and 2t'Im<I> 2 (k) > in A + , 
O.fjk)] 2 = (5-i/2 - 2t'Im$ 2 (k)) 2 + |i$(k)| 2 is 
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smaller than [£(5i,0, -t'\ k)] 2 = (cti/2 + 2^Tm$ 2 (k)) 2 + 
|i$(k)| 2 . Therefore, the convexity d 2 F c g/ 'da 2 becomes 
negative, which disagrees with the assumption that 
(<7i,<72) is the minimum, 
(ii) If (l/a 2 )(dF cS /da 2 )\ 

(6-1,5-2) — 0: we have 



1- / d 2 k- 



1/2 



= 0. 



(B3) 



Using this relation, the derivative in the ci-direction be- 
comes 



dF cff 



, , 2] a 1 /2-2t'Im$ 2 (k) 
= o"i — / a k — 2 



d 2 k- 



a 2 ,t';k)Y 
2t'Im$ 2 (k) 



(B4) 



[S(a u a 2 ,t';k)Y 



which becomes nonzero unless d\ = 0, as shown in the 
case (i). It disagrees with the gap equation in Eq. (fT2|) . 

Therefore, we can conclude that the assumption a =/= 
is incorrect, i.e. the order parameter a is completely 
tilted to the ^-direction. 



Appendix C: Evolution of the order parameter a 

In this appendix, we investigate how the order param- 
eter (ai,a 2 ) evolves as a function of t' and m in detail, 
and discuss how the modified topological phase structure 
is related with that of the noninteracting system. 

Here we fix the spin-orbit coupling amplitude t' and 
vary the staggered magnetic field m. In order to find the 
potential minimum in the (ax, <7 2 )-plane, first we fix <j\ 
and check the sign of 



d 2 k- 



1/2 



[£(ai,a 2 ,t';k)] 



(CI) 



instead of dF c s/da 2 (because F e g(ai,a 2 ) is always even 
in er 2 ). Since the right hand side of Eq. (|Clj) monotoni- 
cally increases and asymptotically reaches toward unity 
as a function of <r 2 (> 0), we have to consider two cases 
depending on its sign at a 2 = 0: 

• (a) If dF c g / d(a 2 )\ a2= o > 0, the effective potential 
is minimized at a 2 = 0. 

• (b) If dF c s/d(a 2 )|o 2 =o < 0, the effective po- 
tential is minimized at finite a 2 , which satisfies 
dF c s/d(a 2 )\a 2 = 0. Here we fix a 2 to be positive. 

We can regard a 2 as a function of a\. The curve com- 
posed of the set of points {(ci, o 2 {a\j)\o\ > 0} in the 
(^l, cr 2 )-plane, which we call here C(i'), is uniquely deter- 
mined by the parameter t' , and continuous in <j\ because 
of the analyticity of Eq. (|Cl|) . 



Along this curve C(t'), the overall potential minimum 
shall be found by varying <j\. By solving the equation 



dF cS 



(o-i ,0-2(0-1)) 



(C2) 



(Ti — 2m 



d 2 k- 



,5- 1 /2-2tTm$ 2 (k) 
[£ (o-i,a 2 (ai),t';k)] 2 



= 0, 



we obtain the potential minimum [d\, d 2 (ai)), as a func- 
tion of the parameters t' and m. 

First, we show that there is a unique one-to-one corre- 
spondence between b\ and m, when t' is fixed to a finite 
value. We can easily see that a\{m = 0) = 0. On the 
other hand, /(cti) asymptotically becomes 01 — 2m as 
a 1 — > 00 (note that a 2 = in this limit), so that we 
obtain the asymptotic solution d\(m — > 00) ~ 2m. 

Now that the boundaries of m and those of are 
matched, we check whether d\ monotonically increases 
between these boundaries as a function of m or not. By 
differentiating both sides of Eq. (|C2p by m, we have the 
relation 



dm- f {ai) 



(C3) 



Thus, what we have to show is that the factor /'(<ti) > 
for any value of 5\(> 0). Here we consider again the 
regions (a) and (b) given above: 

• (a) In the region where <t 2 (<ti) = 0, we have the 
relation 



dF c n 



d(a 2 2 ) 



(*i,0) 



d 2 k- 



1/2 



[£(*!, 0,f;k)]' 



> 0, 



(C4) 



from the definition of this region. This relation 
yields 



f'(*i) =1 - / d 2 k 



1/2 



[£(a u 0,t';k)Y 



(C5) 



d 2 k 



[gi/2-2t / Im$ 2 (k)] 1 
[£(ax,0,t';k)} 4 



> 0. 



• (b) In the region where ct 2 (o"i) 7^ 0, we have the 
relation 



dF ci 



d(a 2 



1- / d 2 k- 



1/2 



q [£(&!, 5- 2 {d-i),t';k)] 



(C6) 



By differentiating both sides of this relation by d\ , 
we obtain a new relation 



,2 k gi/g ~ 2*'Im$ 2 (k) + a' 2 a 2 /2 = 



(C7) 
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FIG. 9: The behavior of the function g(t',ai) for several 
values of t' . This function shows logarithmic divergence at 
<Ji/2 = 3\/3i', where one of the valleys totally loses its spec- 
tral gap. 



where a' 2 = <9ct 2 (ci )/<9<7i . Using these two rela- 
tions, we can simplify /'(di) as 



/'(Si) = 1 



d 2 k- 



1/2 



+ / <i 2 k J 
In 



n [£ (<7i,CT2(<7i),f ;k)] 
h - 2tTm$ 2 (k)] - 2tTm$ 2 (k) + S 2 a 2 /2] 



d 2 k 



[q 1 /2-2t , Ini$ 2 (k)+g 2 g 2 /2] 
[f(a l5 a 2 (ai),t';k)] 4 



> 0. 



(C8) 



Therefore, oi(to) monotonically increases for any to G 
[0,oo). 

Due to this one-to-one correspondence between to and 
(Ti, the solution (ai,<7 2 ) moves continuously along the 



path C(t'), starting from ai(m = 0) = toward 5i — >• oo 
as shown in Fig|4j when f' is fixed and m is varied. Thus, 
as shown in Figj3l the order parameter a\ monotonically 
increases as a function of to, while cr 2 shows a transition 
between c 2 ^ and cr 2 = 0. From here on, we em- 
ploy the "modified" effective mass d\ /2 as the parameter 
characterizing the system, instead of the bare mass m, to 
discuss the phase transition characterized by <r 2 . (When 
a i is given, we can derive the value of to by Eq. (|C2[) .) 

The phase structure of the system is related to the 
behavior of the curve C(t'). The curve leaves from the 
<7i-axis, namely (T 2 ^ 0, in the region (b), which corre- 
sponds to the "tilted antiferromagnetic" phase shown in 
Figf5] On the other hand, the curve coincides with the 
cri-axis in the region (a), which can be classified into the 
conventional or topological insulator phases. According 
to the definition of the regions (a) and (b), the phase 
structure of the system is characterized by the sign of 
the factor 



<?(*', Si) 



dF cS 



d 2 k- 



1/2 



(C9) 

The behavior of g(t', u\ ) for several values of if is shown in 
FiglU This function shows a negative logarithmic diver- 
gence at a/2 = 3y/3t' for any value of t'{^= 0), since one 
of the valleys becomes gapless at this point. Therefore, 
the system under an electron-electron interaction shows 
the tilted AF phase around the topological phase bound- 
ary originally given in the noninteracting system. When 
t' and to become dominant compared to the electron- 
electron interaction, two phase boundaries around the 
tilted AF phase approach asymptotically to the original 
topological phase boundary. 
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